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Abstract —Approximate message-passing (AMP) method is a 
simple and efficient framework for the linear inverse problems. 
In this letter, we propose a faster AMP to solve the Li-Split- 
Analysis for the 2D sparsity separation, which is referred to as 
MixAMP, We develop the MixAMP based on the factor graphical 
modeling and the min-sum message-passing. Then, we examine 
MixAMP for two types of the sparsity separation: separation of 
the direct-and-group sparsity, and that of the direct-and-finite- 
difference sparsity. This case study shows that the MixAMP 
method offers computational advantages over the conventional 
first-order method, TFOCS. 

Index Terms —2D Compressed sensing. Sparse signal separa¬ 
tion, approximate message-passing (AMP), Li-Split-Analysis. 


1. Introduction 

The linear inverse problems for estimating an unknown 
signal X from linear measurements Y are central in signal 
processing techniques. In this letter, we revisit such inverse 
problems with three additives: 

• X is a two-dimensional (2D) square, whose entries are 

with the index pair G A := x 

{1, VN) where VN eZ+ and N := |A|, 

• X is a mixture of two signals Xa,X5 having sparse 
representation in two separate basis, i.e., X = X^ + X5. 

• Y are Compressed Sensing (CS) measurements such that 
Y includes its effective samples with the index pair 
(kd) encA where M := \n\. 

From the 2D setup, we generate the linear measurements by 

Y = Pn {AXA^} e 

where A G K'Av>^'Av jg ^ measurement matrix; Vn{-} 
is the undersampling operator nulling entries of its matrix 
argument not in the set Q. The measurement model O is 
motivated by the 2D signal acquisition concept considering 
two perpendicular spatial/spectral axes of X where “the left 
A” linearly mixes X in vertical axis and “the right A^” does 
in horizontal axis. As practical applications of 0, the 2D 
Fourier-transform-based image/video acquisition | 8|, ca and 
optical image encryption m have been considered. 
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The measurement model 0 has two advantages in 2D 
signal processing. This 2D model spends complexity 

for the measurement generation, which is less than 0{N‘^) by 
the ID equivalent model m, 

vec(Y) = Vq' {(A O A) vec(X)} G (2) 


where ( 8 ) is Kronecker product, vec(-) is the columnwise 
vectorizing operator, and O' C {1,..., A}. In addition, the 2D 
model is storage efficient by saving memory for the Kronecker 
product matrix A (8) A G which is unavoidable under 

the ID model 

Our main task is to simultaneously estimate the two 2D 
signals, X^, X5, given the knowledge of Y, A, and O under 
0. This problem so-called Sparse signal separation, being 
related to the Analysis CS whose unknown signal is sparse in 
a concatenation of the two basis m, including applications to 
image inpainting and deblurring m, and super-resolution El. 

When the cardinality of the sets has \ft\ < |A| such that the 
problems are ill-posed, optimization methods with regulariza¬ 
tion have been mostly considered. A known approach to the 
problem is the Li-Split-Analysis El-llll: 


min IITaXalli + lirbXblli 
s.t. Y = Pn{A(Xa + Xb)A^}, 


( 3 ) 


where the sparsity of X^, X5 is promoted in terms of analysis 
transform operators 7^, T^. This Li-Split-Analysis is related to 
Morphological Component Analysis (MCA) iia.ca in that 
the both analysis approaches decompose X by pursuing sparse 
representation of Xa,X5. However, they are distinct in that 
MCA does not include measurement compression such as 0 
but simply goes with Y = X^ + X5. 

Practical solving of 0 has been considered in the works 
of 111,1151 via the Templates for First-Order Conic Solvers 
(TFOCS) m and the Split Bregman iteration |0, respectively. 
However, their examples are not in the context of our problem 
setup since they do not contain the concept of linear mixing 
by setting A = I. 

In this letter, we propose an Approximate Mess age-Pas sing 
(AMP) method for solving the Li-Split-Analysis under the 2D 
CS model 0. This is motivated by excellent properties of 
AMP ll9l, |[T0l : i) asymptotic Lasso performance, ii) efficient 
computations, and in) algorithmic simplicity. We refer to the 
proposed method as separation of sparse mixture via approx¬ 
imate message-passing (MixAMP). We claim that MixAMP 
is remarkably faster than the conventional first-order method, 
TFOCS ( 3 , for the 2D sparse signal separation task. 

We believe that another advantage of this MixAMP lies in 
its fiexibility. The 2D separation problems, regarding various 
types of the sparsity, can be solved via the MixAMP once 
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Fig. 1. Factor graphical model and message-passing for the Li -Split-Analysis 
problem 

a proper denoiser is given. In the sequel, we examine the 
MixAMP for two cases of the sparsity separation, i) separation 
of direct-and-group sparse mixture, and ii) that of direct-and- 
finite-difference (FD) sparse mixture, by applying the simple 
denoisers introduced in the literature ii.coi. In each case, 
we demonstrate the low-computationality of MixAMP by an 
exemplary comparison to the TFOCS method^ 

IF AMP Method for Li-Split-Analysis 


Algorithm 1 MixAMP Method 

Require: Measurement matrix A G , Measurements Y G , 

Undersampling operator Vq{-} 

Ensure; Recovered signals G 

Initialization: = Y, ^ ||Y||^ 

Xr° = 0,X^,=o = 0 

for t = 1 to t* do do 

X* =7,a(ArR*-lA + X^l;0*-l) 

X‘ = %(A^R‘-iA + X‘-i;0‘-i) 

R* = Y - PniAiXi + X*)A^} 

+R*-i § (W(A^R-*“^A + Xyi;0*-i)) 

+R*-i A <^V(A^R*"^A + X*-i;0*-i)) 

nt _ 1 11 'D 111 2 


where {x} indicates the set of elements in the vector argument 
X. As illustrated in Fig[^ the min-sum rule ^ disjointly 
exchanges the messages between onefactor layer and two dif¬ 
ferent variable layers where the factor nodes are effective only 
with the index {k,l) G ft, and the variable nodes correspond 
to the index {i,j) G A. Then, totally, 4MA^ messages are 
handled per iteration since there are M effective factors, each 
of which generates 2N messages, and 2N variables, each of 
which produces M messages. 

At the fixed-point, the scalar MAP estimate of each signal 
is approximated by 


The first step of the AMP development is to establish a 
message-passing rule based on a factor graphical model of 
linear systems. Figj^ describes a factor graph for our system 
IB’ deriving the corresponding joint PDF, given by 

/x„,X6.y(-) A /x„(';T.)/x6(-;76) II /Yfc,|X„,X6(-) (4) 

where the two priors, /xa(s7^) /x^CsTb), sparsify the 
signals in the domain of Ta and Tb, respectively. From the 
joint PDF 0, we form a message-passing rule for solving 0 
in the manner of the min-sum algorithm ifTTIl . given by 


a) For the signal : V(ji, kl) e A x Q 

min log|X q^ ,X 5 (■) T ^ ^j'i'^kl{^a,j'i') 


\^a,j 


3 * 


+ E 

jieA 

— log /Xa 5 Lt) + ^ ^ t^k'I' 

k'l'^kl 

b) For the signal : V(ji, kl) e A x Q 

Uki^jAxbji) = min log|x^,X 5 (•) + aji^kl{xb,ji) 




jieA 


^ji^kl{Xb,ji) = f:S.i,{^b]'Tb) ^ Uk'l'^jiixbji): 

k'l' ^kl 

(5) 


^The MATLAB codes for this comparison is available at our website, 
https ://sites. google .com/site/j wkang 10/. 


^a,ji 


= arg min aji{xa,ji), Xbji = arg 


min Bji{xb,ji), 


( 6 ) 


where the functions aji{xa,ji),I3ji{xb,ji), which include the 
posterior information, are given by 

~ Ids/X a (^0 5 4" ^ ^ Cl) 

ik,i)en 

I3ji{xb,ji) = logfxt{^b;Tb) + y] i/ki^ji{xb,ji). (8) 

{k,l)eQ 

In the AMP literature (3- im, the second term of 0,(18) 
are handled as Gaussian exponents. This is based on 
the assumption that the factor graph connection, by the 
matrix A, is sufficiently dense such that the messages 
fiki^ji{xaji),aki^ji{xbji) from each factor are Gaussian 
distributed by the law of large numbers. 

The remaining steps for the AMP development consist of 

1) the quadratic approximation step, which approximates 
the min-sum equations by quadratic functions, then con¬ 
verting 0 to a parameter-passing rule whose messages 
are simple real numbers (instead of functions), 

2) the first-order approximation step, which cancels in¬ 
terference caused by the loopy graph connection, and 
reduces the number of messages from 4MA^ to M^2N. 

These development steps are conventional for the AMP meth¬ 
ods applied to the other types of linear inverse problems 
m-iiiii, which are well formulated in the literature CD. 
Therefore, we omit details of such remaining steps in this 
letter, immediately providing a final form of MixAMP in 
Algorithm 
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One thing noteworthy is that in Algorithm the two 
disjoint iterations, described in Figj^ share the residual term 
G which significantly reduces the number of the 

messages in the iteration. This can be explained using the 
two arguments given in CD: i) The messages, sent by the 
(/c, /)-th factor, have a residual form through the quadratic 
approximation step; for example, the message toward the 
variable index (a, ji) is expressed as 

'^a,kl^ji — Vkl ^ft { / Ojf j' ^a,j'i'^kl 

} • (9) 

ii) We can drop the directional dependency upon the destina¬ 
tion index (a, ji) in © by decomposing the residual message 
^a.ki^ji ^ form of “pure residual directional correction” 
under the large limit M, ^ oc, i.e., 

'^a^kl^ji ~ 'b'kl “ 1 “ ^'^a,kl^ji 

-V-^ 

=rki 

CLijCLjiXd^ji^ ( 10 ) 

"-V-" 

= ^ra,kl^ji 

where the correction term dva^ki^ji has the order of 
The above two arguments equivalently hold for 
the factor messages toward the index Therefore, the 

pure residual G is independent of the index of the 
destination variable, enabling the residual sharing in the two 
disjoint iteration. 

The Mix AMP incorporates two distinct denoisers, denoted 
by according to sparsity types in the mixture 

Xq+X^. These denoisers undertake the sub-optimization tasks 
given in ini, generating the MAP estimate X^, X^ at every it¬ 
eration. Hence, choice of the sparsifying priors, /x^ {'), /xf, (*)’ 
determines functional form of the denoisers; for some choices, 
we may need to utilize external numerical solvers for the 
denoiser implementation due to analytical difficulties of the 
prior exponent e.g., non-scalability and non-smoothness. This 
denoising concept of MixAMP is analogous to the shrinkage in 
the context of iterative shrinkage-thresholding (1ST). However, 
they are different in that the MixAMP denoisers shrink X^, X^ 
in the domain of Ta and Th respectively, whereas the shrinkage 
operator of 1ST takes soft-thresholding in the standard domain. 


III. Case Studies for 2D Sparsity Separation 

This section provides an exemplary comparison of the 
MixAMP and the TFOCS Q methods for two different cases 
of the sparsity separation. In each case, we first introduce de¬ 
noisers applied to the MixAMP method, and then we provide 
the separation example. We inform that the TFOCS method 
is not directly applicable to the 2D model ([T]), which should 
be accompanied by the vectorization of This comparison 
is based on the standard Gaussian matrix A whose entries 
are i.i.d. drawn from A/’(0, ^), and MATLAB 2014a with a 
2.67-GHz Intel Quad Core i5 was used to generate the results. 
In addition, we stop the MixAMP and TFOCS iteration when 

tol := < 5 X 10-4 


VlIXalll + IIXjlll 


is met. 


Sparse Mixture Xa+Xb 



20 40 60 80 100 120 


70% Sampling for Y 

5% sparsity of Xa 

II 

U\ 

X 

o 

4 


PSNR 


Solver 

of Xb 

Runtime 

Type 

<dB> 

<sec> 

MixAMP 

26.83 

2.02 

TFOCS 

26.69 

74.20 


MixAMP Separation of Xa 



20 40 60 80 100 120 


TFOCS Separation of Xa 



20 40 60 80 100 120 



Fig. 2. An exemplary comparison of MixAMP and TFOCS in the direct-and- 
group sparsity separation where we reconstruct a 128 x 128 QRcode image 
Xfe by removing shot noise (5% sparsity) given the measurements Y with 
70% sampling (M/N = 0.7). Here, we set e = 10“^°, Ai = 0.5, A 2 = 1.2 
for the TFOCS method (H 


A. Separation of Direct and Group Sparsity 

We consider separation of a 2D direct-and-group sparse 
mixture. For the direct sparsity denoiser, we apply the soft- 
thresholding which has been the most widely used because of 
its simplicity Let X^ G have the direct sparsity. 

Then, we can estimate Xaji G M via a scalable denoiser 
77a(•) : M —> M, given by 

Xa,ji = Va{x; 9) := arg min + fXaji - xf 

= sgn(a;) max{|a;| — 0 }. (11) 

In ([TT](, the input x G M is a Gaussian variable corresponding 
to the second term of the posterior function 0. This Gaussian- 
ity let the denoiser to solve a penalized least squares problem, 
which is common in the AMP denoisers. In addition, it is well 
known that the soft-thresholding is based on the Laplace prior; 
therefore, we have log/x„(xa) = ||Xa||i in 0. 

We adopt the block soft-thresholding for the group spar¬ 
sity pursuit fiol . Let Xft e RV^XV^ have the group 
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sparsity. Then, the block soft-thresholding denoiser, 7 ^ 5 (•) : 
is given by 

X6.B = rib{xB-,0) := arg min ||X6 ,b||f + ^||X6,b - :x-b\\f 

= xb • max{l - 6 >||xb||^\0} (12) 

where the input x^ G is a block matrix such that the 

signal X 5 is partitioned into N/B square blocks with the size 
B. Specifically, this block denoiser makes its block argument 
x^ to a zero matrix if ||x^||i? < 0, otherwise diminishing 
it by the quantity 0 to the origin. In addition, this block 
thresholding is related to a block Gaussian prior; hence, we 

have log/x,(xb) = Eaii blocks I|X 6 .b||f from 

Fig|^ displays a separation example by Mix AMP and 
TFOCS, where we reconstruct a 128 x 128 QRcode image 
X 5 by removing shot noise X^ given the measurements Y 
with 70% sampling {M/N = 0.7). The QR code refers to the 
group sparsity part of the mixture image, while the shot noise 
models the direct sparsity part. In the TFOCS method, we 
recast 0 to Combining-Li-and-Group minimization, solving 



Sparse Mixture Xa+Xb 


20 

40 

60 

80 

100 

120 


50% Sampling for Y 

10% sparsity of Xa 

m/ = 5xi0“" 

Solver 

Type 

PSNR 
of Xb 
<dB> 

Runtime 

<sec> 

MixAMP 

23.16 

59.74 

TFOCS 

23.68 

122.50 


MixAMP Separation of Xa 



20 40 60 80 100 120 


TFOCS Separation of Xa 



20 40 60 80 100 120 


min Ai||Xa||i+A 2 ^ 

All blocks (13) 

s.t. ||Y-Pn{A(X„+X 6 )A^}||^<£ 

where 5, Ai, A 2 >0 are calibration scalars. The result of Fig|^ 
reports that MixAMP is much faster than TFOCS in CPU 
runtime while providing comparable reconstruction quality in 
PSNR. 


B. Separation of Direct and Finite-Difference Sparsity 


MixAMP Separation of Xb 
20 

40 
60 
80 
100 
120 

20 40 60 80 100 120 



TFOCS Separation of Xb 



20 40 60 80 100 120 


We present another case by introducing finite-difference 
(FD) sparsity, then addressing a separation problem with a 
direct-and-FD sparse mixture. For the FD sparsity pursuit, 
we apply the total variation (TV) denoiser Eol which has 
been investigated by the numerous literature (see for example 
(121). The TV denoiser is neither scalable nor block-separable 
because the FD sparsity cannot be defined by a single scalar 
of the signal, but depending upon all the adjacent of the scalar. 
Let X 5 G MxWxvW pj^ sparsity. Then, we consider 

the TV denoiser, 775 (•) : ^ which solves 

Xb = r? 6 (x; A) := arg min |[Xb] |tv + ^||Xb - x|||.. (14) 

X 5 z 

It is recognizable from ^ that the TV norm HX^Htv is the 
exponent of the sparsifying prior such that log/xf,(x 6 ) = 
IIX^IItV" Foi* the implementation of ( p^ , external numerical 
solvers have been mostly considered since the TV norm is 
analytically non-scalable and non-smooth ca. 

In Fig[^ we simultaneously estimate a shot noise X^ (10% 
sparsity) and a 128 x 128 Cameraman image X^ from the 
2D measurements Y with 50% sampling {M/N = 0.5). In 
this example, we postulate that the Cameraman image has 
the FD sparsity and the shot noise is directly sparse. For 
the MixAMP separation, we apply the soft-thresholding O 
for Xa, implementing the anisotropic TV denoising for 
X 5 using the 2D-Bregman iteration (Section 4 of 111). In 


Fig. 3. An exemplary comparison of MixAMP and TFOCS in the direct- 
and-FD sparsity separation where we simultaneously estimate a shot noise 
Xa (10% sparsity) and a 128 x 128 Cameraman image from the 2D 
measurements Y with 50% sampling (M/N = 0.5f Here, we set 6 = 
10 - 10 , ^ 2 . 0 , A 2 = 1.4 for the TFOCS method (Ts). 

the TFOCS method, we recast © to Combining-Li-and-TV 
minimization, solving 

min Ai||Xa||i + A 2 IIX 5 IITV 

s.t. ||Y-Pn{A(Xa + Xb)A^}||F <£. 

Likewise to the example in Section III-A, the result of Figj^ 
validates the computational advantages of MixAMP where 
MixAMP is approximately 2 time faster than TFOCS for 
the same task. The scale difference of the runtime gap from 
the result of Section III-A is coming from the fact that the 
Bregman-TV denoiser (El requires more computations than 
the block soft-thresholding ( p^ does. 

These two comparison results support our claim that Mix¬ 
AMP outperforms the TFOCS method in computational effi¬ 
ciency under the 2D sparse signal separation task. 

IV. Conclusions 

In this letter, we have discussed the MixAMP method for 
the ill-posed Li-Split-Analysis, applying the 2D sparse signal 
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separation problem. We first have developed the MixAMP 
method based on the factor graphical model of the 2D CS 
measurement model Q- Then, we have provided two cases of 
the study for the sparsity separation, validating the computa¬ 
tional advantages of MixAMP through exemplary comparisons 
to the conventional first-order method, TFOCS |I71. Therefore, 
we claim that MixAMP is a very good alternative of the 
TFOCS method for solving the Li-Split-Analysis in the 2D 
sparse separation problem. 

Supplementary Material 

A. 2D Compressed Sensing Model Q.a Practical Viewpoint 

In this letter, we consider “the compressed sensing (CS) 
technique” with the model 0 . The model 0 requires two- 
steps for the CS measurement generation of Y as shown in 
Fig|^ 1 ) linear mixing in two perpendicular axes by AXA^ 
and 2) undersampling hy Vq{'}. One can argue that this two- 
steps generation makes us to lose an advantage of CS which 
generates Y in the first place. Nevertheless, we state that this 
two-step generation is useful because of two practical reasons: 

1) For fast measurement generation with unitary matrices 

A: The model 0 can provide an accelerated gen¬ 
eration of Y when the matrix A is some unitary 
types. For instance, this acceleration utilizes Fast Co¬ 
sine Transform (FCT) when A is the DCT matrix, 
and Fast Fourier Transform when A is the Fourier 
matrix. Here, we consider the DCT case as an example. 
Let Adct ^ DCT matrix and 

AsubDCT ^ ]^v2vtxv^ ^ sub-DCT matrix whose rows 
are randomly sampled from Adct- The conventional CS 
generation, expressed as 

Y = AsubDCT X A^„bDCT e (16) 

spends 0{NM ^) computations due to the matrix multi¬ 
plications. Compared to ( p^ , the FCT-based generation 
with the model 0, Le., 

Y = Po{AdctXA5ct}, 

= Pn{FCT2D[X]} e (17) 

is computationally efficient with 0{N log N^); there¬ 
fore, its efficiency gets remarkable as the system size N 
increases. In addition, it is obvious that the FCT method 
is not applicable with the sub-DCT matrix AsubDCT- 

2) Use of a random decimation operator instead ofVn{’}'- 
In practice, the random undersampling operator Vo{'} 
can be simply replaced by a random decimation operator. 
Then, the samples of Y can be stored to a memory 
with the size M by holding the knowledge of the set 
Vl. In this case, however, we need an inverse operator 
of the decimation operator to calculate A^RA for the 
MixAMP method. 


This is because the sparsity separation includes not only 
the reconstruction of the mixture signal X from the CS 
measurements Y, but also the separation of the two sparsity, 
Xa and X 5 , from the mixture X. We can look at this hardship 
of the sparsity separation by simple manipulation from ([T]): 

A^j, ( 18 ) 

From ( p^ , we can reasonably conjecture that the reconstruc¬ 
tion of the concatenated signal [X^jX^]^ from Y requires 
larger M than the single sparsity recovery does. In addition, 
while revising this letter, we became aware of a theoretical 
work of Studer et al. ca which supports our conjecture by 
providing a coherence-based sufficient condition for recovery 
guarantees of X^ and X^. 

Naturally, the reconstruction quality of the sparse separation 
gets better as the sampling rate M/N increases. In Fig|^ 
we provide an extended result of the X 5 reconstruction in 
the directed-and-FD sparsity separation (Section III-B) for a 
variety of M/N. From the figure, we see that the both methods 
improve their reconstruction quality given a higher M/N. 
This result also shows that although the TFOCS method has 
a small lead in the reconstruction quality as M/Y becomes 
higher, the MixAMP method is still remarkably advantageous 
in computational cost. 


Y = Vn{ [A|A] 


Xa 

Xb 


B. Hardship of Sparse Separation Problem 

The sparsity separation, handled in this letter, is basically 
harder problem than the conventional single sparsity recovery. 
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Undersampling 
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Fig. 4. An example of 2D CS measurement generation in the model (1), where we consider the DCT matrix A and the 128 x 128 Cameraman image as 
the 2D target signal X. First, the signal X is linearly mixed in two perpendicular axes by the DCT matrix A, generating AXA"^. Second, the measurement 
Y is produced by applying the undersampling operator 7 ^q{-} to AX A"*". 


MixAMP 


PSNR 

Runtime 


TFOCS 


PSNR 

Runtime 


M/N=0.5 M/N=0.6 M/N=0.7 M/N=0.8 



Fig. 5. Extended numerical result of the directed-and-FD sparsity separation in Section III-B: In this figure, we only plot reconstruction of the 128 x 128 
Cameraman image X^ for a variety of the sampling rate M/N. The others for the experimental setup remains the same as Section III-B. For the TFOCS 
method (15), we use empirically-tuned parameter set (Ai, A 2 , e). 
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